DM-54894: Add new utilities from analysis notebooks and update existing plots#64
Open
isullivan wants to merge 30 commits into
Open
DM-54894: Add new utilities from analysis notebooks and update existing plots#64isullivan wants to merge 30 commits into
isullivan wants to merge 30 commits into
Conversation
7dc3556 to
1e0279e
Compare
301f00f to
2aaff4d
Compare
Also clean up the remaining docstrings
Read integer columns through the SDM schema so nullable BIGINTs round-trip exactly instead of being silently coerced to float64 by pd.read_sql_query. Fall back from validityEndMjdTai to the legacy validityEnd name for older APDB fixtures.
The metadata rows and flag-tag overlays are now declarative tables driven by a shared loop, instead of a long sequence of inline fig.text calls. A new heights kwarg lets subclasses with extra panels position the rows themselves.
New compare module: match_catalogs cross-matches two DiaSource frames within optional group columns, flux_residuals computes per-band/per-flux residual statistics for matched pairs, and match_to_truth attaches the nearest truth row to each detection.
display_images is rewritten around a shared overlay-collection helper: catalogs are loaded once, color-coded by flag group when requested, and the same overlays are reused for every frame. New display_images_ab puts two butlers side-by-side, and extract_timestamped_messages flattens an LSST JSON log.
Three helpers: lightcurve plots a per-band psfFlux history for a single diaObject (with optional forced-photometry overlay), cutout_grid lays out science/template/difference cutouts for many sources in a mosaic, and summarize_run produces a per-visit summary of an APDB run.
Extends PlotImageSubtractionCutoutsTask with a lightcurve panel beneath the science/template/difference cutouts. The panel shows the diaSource history for the associated diaObject and overlays forced-photometry measurements for visits without a detection.
Walks the per-(visit, detector) DiaSource/DiaObject/DiaForcedSource datasets that DiaPipelineTask writes alongside its APDB inserts and concatenates them into single DataFrames matching the APDB SDM schema. Useful for QA without round-tripping through a live APDB.
image_diff_stats reports median, MAD, stdev, kurtosis, and mask-plane fractions for a single difference image. pixel_compare aligns two difference images on the same dataId and returns their pixel difference, ratio, mask XOR, and summary statistics.
subtraction_quality_report consumes a per-detector *_spatiallySampledMetrics table and prints percentile summaries plus a three-panel diagnostic figure for diffim_chi2PerPix, psfMatchingKernel_residualNorm, and dipole_density.
Walks every <task>_metadata dataset in a butler collection, extracts timing fields via QuantumResourceUsage.from_task_metadata, applies a per-task threshold (keep the whole task if any quantum exceeds it), and returns a tidy DataFrame with an optional box plot.
`find_objects_sharing_sources` walks the (diaSource, run-1 diaObject, run-2 diaObject) graph to return the full connected cluster reachable from a starting diaObjectId. `classify_association_clusters` enumerates every such cluster across two runs and labels each as matched, split, merged, or tangled.
`plot_cutouts_with_object_markers` renders per-diaSource cutouts with color-coded diaObject markers and "other diaSource" overlays. `plot_objects_sharing_sources` is a two-column wrapper that compares the run-1 and run-2 views of the cluster reachable from a starting diaObjectId.
In plot_objects_sharing_sources, source markers now take the palette color of the diaObject they belong to instead of a single shared color. The right panel maps sources to their run-2 owner so the same source can read differently across the two runs.
2aaff4d to
218274d
Compare
cd998ed to
5a2080a
Compare
… visible. Also ensure that other markers are drawn in pipeline order, so that the last version is the one that's visible.
BrunoSanchez
approved these changes
Jul 9, 2026
BrunoSanchez
left a comment
Member
There was a problem hiding this comment.
I have read all changes, I am on my way now to test some of these in a notebook with some Rubin alert data, or CI package.
I have run your notebooks but they don't show all the functionality added here.
I approve the changes, but will keep commenting. I think it is good to merge, but building more examples can be useful, and help also test this a bit more.
| from .spatiallySampledMetricsQA import * | ||
| from .plotUtils import * | ||
| from .taskRuntimes import * | ||
| # NOTE: do not import from nb_utils in this file, as it depends on packages |
Member
There was a problem hiding this comment.
What about this? You are adding an from .nb_utils import *
| return unique1, unique2, matched | ||
|
|
||
|
|
||
| def find_objects_sharing_sources(diaObjectId, sources1, sources2, |
Member
There was a problem hiding this comment.
I haven't yet tested this, but it is a very interesting functionality.
Comment on lines
+49
to
+57
| BAND_COLORS = { | ||
| "u": "C4", | ||
| "g": "C2", | ||
| "r": "C1", | ||
| "i": "C3", | ||
| "z": "C5", | ||
| "y": "k", | ||
| } | ||
|
|
Member
There was a problem hiding this comment.
Why not loading the lsst official color palette?
The existing erase call only erased the image, not any markers drawn on top of it.
361f82f to
254eaf1
Compare
9793747 to
47113a4
Compare
47113a4 to
eecb6c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.